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Abstract 

The  Department  of  Defense  (DoD),  due  to  inconclusive  results  from  DoD  source  selection 
evaluation  teams  (SSETs),  is  vulnerable  to  selecting  contractors  that  are  not  offering  the  best 
value  to  the  government.  Cryptic  SSET  results,  furthermore,  render  the  DoD’s  acquisition 
process  susceptible  to  corruption,  thus  endangering  the  DoD’s  reputation  as  well  as  the 
careers  and  freedom  of  DoD  military  and  civilian  personnel.  The  DoD’s  source  selection 
directives  in  DFARS  Subpart  215.3,  Source  Selection,  appear  to  be  skillfully  and 
professionally  written.  In  actuality,  however,  there  are  weaknesses  that  contribute  to  cryptic 
SSET  results.  The  DoD’s  prohibition  against  numerically  weighing  proposal  evaluation  factors 
and  prohibition  against  assigning  numerical  scores  to  subjectively  rated  factors  contribute  to 
ambiguous  SSET  results,  provide  insufficient  transparency,  and  render  the  process 
susceptible  to  fraud.  This  paper  describes  instances  of  contract  corruption  by  government 
officials,  provides  accolades  and  criticisms  of  the  DoD  source  selection  principles  and 
procedures,  and  recommends  changes  to  clearly  identify  contractors  offering  best  value  to 
the  government.  Implementation  of  the  recommendations  presented  here  will  remove 
ambiguity  from  the  SSET,  improve  transparency,  and  reduce  opportunities  for  corrupt  officials 
to  prosper  through  nefarious  acts. 

Introduction 

The  Department  of  Defense’s  (DoD’s)  principles  and  procedures  for  selecting 
contractors  when  conducting  negotiated,  competitive  acquisitions  are  contained  in  DFARS 
subpart  215-3,  Source  Selection.  The  scope  of  subpart  215.300  refers  contracting  officers  to 
the  provisions  of  a  memorandum  from  the  Director,  Defense  Procurement  and  Acquisition 
Policy  (Director)  for  conducting  such  acquisitions.  The  memorandum,  dated  March  4,  201 1 , 
has  the  subject  line  “Department  of  Defense  Source  Selection  Procedures.”  The  mandatory 
principles  and  procedures  contained  in  this  memorandum,  as  expected,  appear  to  constitute 
a  well  conceived  process  for  evaluating  contractor  proposals  received  in  response  to 
requests  for  proposals  (RFPs).  The  purpose,  roles,  and  responsibilities  for  acquisition 
officials  are  well  defined  and  clearly  stated.  Pre-solicitation  activities  are,  likewise,  logical 
and  clearly  communicated.  One  recommended  additional  provision  regarding 
communications  during  the  pre-proposal  phase  of  the  contracting  cycle,  however,  is  offered 
in  this  paper.  The  proposal  evaluation  and  decision  process  required  by  provisions  of  the 
Director’s  memorandum  also  portray  the  initial  impression  of  judicious  design  to  guide  the 
SSET  in  selecting  the  contractor  offering  the  best  value  to  the  government.  A  critical 
evaluation  of  the  process  and  restrictions  in  the  manner  for  scoring  proposals,  however, 
reveals  that  the  provisions  are  susceptible  to  the  formulation  of  cryptic  SSET  results,  less 
than  the  desired  degree  of  transparency,  and  corruption  of  the  contractor  selection  process. 
This  paper  highlights  commendable  aspects  of  current  policies  and  procedures,  provides 
examples  of  contract  malfeasance  to  demonstrate  the  existence  of  chicanery  in  government 
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contracting,  illustrates  weaknesses  in  present  policies  and  procedures,  describes  the 
recommended  process  for  evaluating  proposals,  illustrates  how  implementation  of  the 
recommended  evaluation  method  will  identify  the  contractor  offering  the  best  value,  and 
offers  additional  recommendations.  The  appendices  to  the  Director’s  memorandum, 
Appendix  A,  Lowest  Pride  Technically  Acceptable  Source  Selection  Process,  and  Appendix 
B,  Debriefing  Guide,  are  not  addressed  in  this  paper. 

Commendable  Aspects  of  Current  Practices 

The  direction  provided  to  DoD  personnel  for  evaluating  proposals  and  selecting  the 
contractor  offering  the  best  value  to  the  government  is  professionally  written  and  excels  in 
its  statement  of  goals  and  the  qualifications  for  personnel  involved  in  source  selection 
activities.  The  roles  and  responsibilities  of  the  parties  participating  in  the  selection  of  the 
best  qualified  contractor  are  clearly  and  intelligently  prescribed.  The  explanation  of  activities 
undertaken  prior  to  issuance  of  a  solicitation  to  prospective  contractors  is  exceptional. 
Although  there  is  no  discussion  of  activities  following  release  of  the  solicitation  and  the 
receipt  of  proposals,  the  discussion  of  the  evaluation  of  proposals  is,  again,  thorough  and 
clear.  Direction  is  provided  for  every  party  involved  in  the  contractor  selection  process. 
Exception  is  taken  here  to  the  method  for  weighing  the  importance  of  factors  and  subfactors 
(future  reference  to  factors  should  be  considered  as  reference  to  both  factors  and 
subfactors)  as  well  as  to  the  prescribed  methodology  for  scoring  the  proposal  evaluation 
factors.  Despite  the  exceptions  taken  here  to  certain  DoD  contractor  selection  policies  and 
procedures,  the  overall  content  of  the  directions  for  selecting  DoD  contractors  is  superb. 

Examples  of  Contract  Malfeasance 

Instances  of  DoD  military  and  civilian  personnel  becoming  embroiled  in  contract 
corruption  have  appeared  repeatedly  in  the  media  for  decades.  One  recent  contract 
corruption  investigation  involving  the  selling  of  classified  information  for  sex,  money,  and 
other  gratuities  resulted  in  the  contractor,  Glenn  Defense  Marine  Asia,  allegedly  being 
positioned  to  overcharge  the  Navy  millions  of  dollars  for  ship  repairs  (Whitlock,  2013).  The 
account  of  the  investigation  indicates  that  despite  the  Singapore  Senate’s  criticism  of  the 
contractor  for  dumping  untreated  sewage  near  Subic  Bay  and  competing  contractors 
complaining  for  years  to  the  Navy  about  unfair  business  practices,  Glenn  Defense  Marine 
Asia  was  awarded  a  $1  million  no-bid  contract  extension.  The  investigation  revealed 
weaknesses  in  the  Navy’s  contracting  practices  worldwide.  Competing  contractors 
complained  that  Glenn  Defense  Marine  Asia  quoted  prices  so  low  that,  in  addition  to  not 
making  profit,  the  prices  would  not  have  covered  the  contractor’s  expenses.  Contract 
loopholes,  as  reported  in  the  referenced  article,  permitted  the  contractor  to  compensate  for 
low  proposed  prices  by  overcharging  the  Navy  millions  of  dollars  for  maintenance  and 
repairs.  The  overcharging  was  allegedly  facilitated  by  Navy  personnel  who  provided  the 
contractor  with  classified  information  regarding  the  location  of  naval  vessels.  As  of  the  date 
of  the  article,  seven  senior  Navy  officials  were  charged,  suspended,  or  placed  on  leave.  The 
Navy  reportedly  expects  additional  personnel  to  be  disciplined  in  this  case. 

One  of  the  more  disturbing  contract  corruption  cases  involved  one  of  the  Navy’s 
highly  decorated  heroes  of  the  Vietnam  War  (Spagat,  2013).  Following  his  distinguished 
naval  career  which  included  becoming  the  first  ace  of  the  Vietnam  War,  being  awarded  the 
Navy  Cross,  a  Silver  Star  with  one  oak  leaf  cluster,  and  the  Purple  Heart,  he  served  as  a 
member  of  the  U.S.  House  of  Representatives  from  1991  to  2005.  Randall  Cunningham, 
however,  pled  guilty  to  steering  government  contracts  to  contractors  in  exchange  for  bribes 
including  a  luxury  house,  Rolls  Royce,  and  other  expensive  gifts.  One  former  contractor, 
Brent  Wilkes,  was  convicted  of  bribing  Cunningham  with  cash  and  gifts  valued  at  over 
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$700,000  in  exchange  for  almost  $90  million  in  DoD  contracts.  The  AP  report  indicates  that 
Wilkes  is  not  serving  time  while  he  awaits  an  appeal  of  his  conviction.  Another  former 
contractor,  Mitchell  Wade,  served  time  in  prison  after  pleading  guilty  to  providing 
Cunningham  with  over  $1  million  in  gratuities,  including  a  yacht,  in  exchange  for 
approximately  $150  million  in  government  contracts.  Although  the  value  of  the  gifts 
Cunningham  received  from  the  two  former  contractors  was  approximately  $1 ,700,000,  that 
amount  was  eclipsed  by  the  $240  million  in  contracts  that  became  a  liability  for  taxpayers. 
This  taxpayer  obligation  is  especially  egregious  when  one  considers  that  during  sentencing 
arguments,  Assistant  U.S.  Attorney  Phil  Halpern  stated  that  while  Cunningham  was  living 
the  good  life,  “he  ...  was  squandering  precious  tax  dollars  for,  among  other  things,  systems 
the  military  didn’t  ask  for,  didn’t  need  and  frequently  didn’t  use”  (“Ex-Congressman,”  2006). 

Weaknesses  in  Policies  and  Procedures 

In  preparation  for  writing  a  book  on  government  contracting  (Curry,  2014),  protests 
over  a  two-year  period  that  were  sustained  by  the  Government  Accountability  Office  (GAO) 
were  studied  to  determine  the  rationale  for  sustaining  the  protests.  Knowledge  of  the  bases 
for  the  GAO  sustained  protests  is  key  to  identifying  reasons  for  the  federal  government’s 
failure  to  consistently  meet  its  goal  (FAR,  2014)  to  select  the  proposal  offering  the  best 
value  to  the  government.  Infractions  discovered  while  conducting  the  research  are  identified 
below  in  descending  order  of  frequency  (indicated  in  parenthesis): 

•  Errors  Made  in  Proposal  Evaluation  Process  (50) 

•  Deficiencies  in  Determining  Acceptable  Proposal  (16) 

•  Socioeconomic  Contracting  Irregularities  (15) 

•  Irregularities  during  Discussions/Negotiations  (8) 

•  Conflicts  of  Interest  (1 ) 

•  Contract  Award  Irregularities  (1) 

In  addition  to  the  reasons  cited  by  the  GAO,  matrices  reflecting  SSET  results  for  all 
federal  government  agencies  were  observed  in  20  of  the  65  protests  reviewed  during  the 
research  project.  Although  the  acquisitions  were  not  all  performed  by  DoD  agencies,  the 
federal  acquisition  offices  involved  frequently  followed  the  DoD’s  lead  in  displaying  the 
SSET  results  in  matrices  containing  adjectival  or  color  ratings  for  factors.  In  two  of  the  20 
cases,  the  contractor  offering  the  best  value  was  identified.  In  the  other  18  cases,  however, 
the  matrices  displayed  inconclusive  results  regarding  which  contractor  offered  the  best 
value. 


The  two  primary  weaknesses  in  the  DoD’s  efforts  to  identify  the  best  value  proposal 
stem  from  restrictions  against  using  numerical  values  to  score  proposals  and  against 
numerical  values  to  weigh  the  importance  of  proposal  evaluation  factors.  The  argument 
opposing  the  restriction  against  using  numerical  values  to  score  proposals  will  be  divided 
between  factors  that  are  subjectively  rated  and  factors  that  are  objectively  rated. 

With  respect  to  the  restriction  against  using  numerical  values  to  score  subjectively 
rated  factors,  consider  the  matrix,  found  in  the  GAO  (2008)  protest  decision,  reflecting  the 
SSET’s  evaluation  of  proposals  for  the  Air  Force’s  aerial  refueling  tanker,  shown  in  Table  1, 
Actual  Proposal  Rating  for  Aerial  Refueling  Tanker. 
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Table  1.  Actual  Proposal  Rating  for  Aerial  Refueling  Tanker 


Boeing 

Northrop  Grumman 

Mission  Capability/Proposal  Risk 

Key  System  Requirements 

Blue/Low 

Blue/Low 

System  Integration/Software 

Green/Moderate 

Green/Moderate 

Product  Support 

Blue/Low 

Blue/Low 

Program  Management 

Green/Low 

Green/Low 

Technology 

Maturity/Demonstration 

Green 

Green 

Past  Performance 

Satisfactory 

Confidence 

Satisfactory 

Confidence 

Cost/Price  (MPLCC) 

$108  044  Billion 

$108,010  Billion 

Cost  Risk 

SDD  Phase/Production  & 
Deployment  Phase 

Moderate/Low 

Low/Low 

IFARA  Fleet  Effectiveness  Value 

1.79 

1.9 

MPLCC  =  Most  Probable  Life  Cycle  Cost 

IFARA  =  Integrated  Fleet  Aerial  Refueling  Assessment 

SDD  =  System  Development  and  Demonstration 


The  actual  adjectival  and  color  ratings  reflected  in  Table  1  reflect  tied  or  virtually  tied 
scores  for  each  factor  thus  resulting  in  obscure  results  from  the  SSET.  An  examination  of 
the  SSET’s  findings  in  the  previously  cited  GAO  (2008)  decision,  however,  indicates  that  the 
SSET  members  discerned  distinct  differences  in  the  contractors’  proposals  for  the  evaluated 
factors.  Indicators  of  this  discernment  of  differences  in  the  evaluation  of  factors  are  provided 
in  the  statements  below  that  were  taken  from  the  referenced  GAO  (2008)  decision: 

Ultimately,  the  SSAC  [source  selection  advisory  council]  concluded,  however, 
that  Northrop  Grumman’s  proposal  was  more  advantageous  to  the  agency 
than  Boeing’s  under  the  mission  capability,  past  performance,  cost/price,  and 
IFARA  factors;  the  two  firms  were  found  to  be  essentially  equal  under  the 
proposal  risk  factor. 

Northrop  Grumman’s  evaluated  advantage  under  the  mission 
capability  factor  was  largely  based  upon  the  firm’s  perceived  superiority 
under  the  key  system  requirements  and  program  management  subfactors; 
the  two  firms  were  found  essentially  equal  under  the  remaining  three 
subfactors. 

In  the  aerial  refueling  area,  the  SSAC  noted  “major  discriminators”  in 
favor  of  Boeing  under  several  KPP  [key  performance  parameters]  No.  1 
objectives,  including  its  capability  to  [Deleted],  and  for  a  noteworthy  non- 
KPP/KSA  [key  performance  parameters/key  system  attributes]  capability  to 
[Deleted]. 
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The  SSAC  also  noted  a  number  of  “major  discriminators”  in  favor  of 
Northrop  Grumman  in  the  aerial  refueling  area,  including  one  under  the  KPP 
NO.  2  objective  for  Northrop  Grumman’s  proposal  to  exceed  the  RFP’s  fuel 
offload  versus  unrefueled  radius  range  (Boeing’s  aircraft  was  also  evaluated 
as  exceeding  this  KPP  objective  but  to  a  lesser  degree),  ...  and  for  a  number 
of  non-KPP/KSA  requirements,  including  the  proposal  of  a  better  aerial 
refueling  efficiency  (more  pounds  of  fuel  offload  per  pound  of  fuel  used)  than 
Boeing’s;  a  “boom  envelope”  that  was  [Deleted]  times  greater  than  that 
defined  by  the  Allied  Technical  Publication  (ATP)-56-[23j  (Boeing  proposed  a 
boom  envelope  that  was  [Deleted]  times  greater  than  that  defined  by  the 
publication);  and  a  higher  offload  and  received  fuel  rate  than  Boeing.  ... 

In  the  aerial  refueling  area,  the  SSAC  also  identified  five 
“discriminators  offering  less  benefit”  for  Boeing  that  were  assessed  under  14 
different  SRD  [systems  requirement  documents]  requirements  and  one  such 
discriminator  for  Northrop  Grumman  that  was  assessed  under  2  SRD 
requirements.  ... 

The  GAO  decision  continued  by  describing  numerous  other  distinctions  between  the 
Northrop  Grumman  and  the  Boeing  proposals.  Recall,  however,  that  the  matrix  summarizing 
the  scoring  of  the  two  proposals  (see  Table  1)  reflects  virtual  ties  between  the  competing 
contractors  for  all  factors. 

The  present  methodology  restricts  the  rating  for  subjectively  rated  factors  to  an 
adjective,  such  as  excellent  or  outstanding,  or  a  color,  such  as  blue  or  green,  for  all 
proposals  fitting  the  narrative  for  the  particular  color  or  adjective.  Sophisticated  DoD 
contractors  are  likely  to  submit  proposals  meeting  the  highest  standards  for  many,  if  not  all, 
factors.  Although  SSET  members  are  likely  to  discern  a  hierarchy  of  ratings  between  the 
competing  contractors,  they  are  presently  forced  to  assign  identical  adjectival  or  color  scores 
to  each  contractor’s  proposal.  The  tied  scores,  despite  the  discernment  of  such  a  hierarchy 
in  the  merits  of  competing  proposals,  are  demonstrated  in  Table  1. 

During  the  research  into  protests  sustained  by  the  GAO,  discussed  earlier,  it  was 
noted  that  objectively  rated  factors  were  occasionally  given  adjectival  or  color  scores. 

Factors  that  can  be  rated  objectively,  however,  would  be  better  understood  if  the  decision 
matrix  reflected  the  actual  numerical  value  proposed  by  the  contractor  or  as  recalculated  for 
reasonableness  by  the  government.  Another  problem  with  adjectival  and  color  scores 
assigned  to  objectively  rated  factors  is  that  calculations  cannot  be  performed  to  characterize 
the  scores  on  an  equivalent  basis,  including  the  ability  to  convert  low  values  to  high  scores 
for  factors,  such  as  cost/price,  that  are  favorable  with  low  values.  The  value  of  converting 
scores  is  discussed  later  in  this  paper. 

With  respect  to  the  restriction  against  using  numerical  values  to  weigh  the 
importance  of  proposal  evaluation  factors,  consider  the  present  method  for  reflecting  the 
significance  of  factors  by  inclusion  of  a  statement  in  the  solicitation  such  as  the  following 
from  the  GAO’s  decision  regarding  a  protest  of  the  contractor  selection  for  the  Air  Force’s 
aerial  refueling  tankers: 

...  mission  capability,  proposal  risk,  and  past  performance  factors  were  of 
equal  importance  and  individually  more  important  than  the  cost/price  or 
IFARA  [integrated  fleet  aerial  refueling  assessment]  factors,  and  that  the 
cost/price  and  IFARA  factors  were  of  equal  importance.  The  subfactors  within 
the  mission  capability  factor  were  stated  to  be  of  descending  order  of 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-574- 


importance.  ...  Proposal  risk  would  only  be  assessed  at  the  mission  capability 
subfactor  level  and  for  only  the  first  four  subfactors.  (GAO,  2008) 

The  primary  weakness  of  this  statement  of  relative  values  is  that  the  values  for  the 
evaluation  factors  are  subject  to  manipulation  during  the  proposal  evaluation  process.  A 
corrupt  official  might  assign  greater  weight  than  originally  envisioned  to  factors  where  a 
favored  contractor  is  strong  while  lesser  weight  might  be  assigned  to  factors  where 
competing  contractors  are  strong. 

The  restrictions  against  using  numerical  values  to  score  factors  and  to  determine  the 
significance  of  evaluation  factors  place  unreasonable  limitations  on  the  evaluation  process. 
Should  these  restrictions  against  using  numerical  values  be  eliminated,  the  SSET  results 
would  more  clearly  define  the  contractor  offering  the  proposal  representing  the  best  value  to 
the  government,  improve  transparency  in  the  evaluation  process,  and  remove  the  obscurity 
of  SSET  results,  thereby  minimizing  the  introduction  of  opportunities  to  corrupt  the 
contractor  selection  process. 

Recommended  Process  for  Evaluating  Proposals 

Rather  than  resorting  to  the  depiction  of  the  SSET’s  findings  through  a  confusing 
matrix  containing  numbers  that  may  be  favorable  either  if  high  or  low,  colors,  or  adjectives, 
identification  of  the  contractor  offering  the  best  value  is  recommended  through  calculation  of 
a  combined  numerical  score  that  distinctly  identifies  the  contractor  proposing  the  best  mix  of 
factors  meeting  the  government’s  needs.  Just  as  with  the  present  process  for  rating 
proposals,  however,  it  is  necessary  to  first  determine  what  factors  will  be  measured  and  how 
they  should  be  measured.  Continuance  of  the  present  practice  of  documenting  the  rationale 
for  ratings  assigned  to  factors  contemporaneously  with  the  evaluation  of  proposals  is  also 
recommended. 

The  memorandum  from  the  Director,  Defense  Procurement  and  Acquisition  Policy 
(Director),  referenced  in  the  Introduction,  prohibits  numerical  scoring  of  proposals  and 
numerical  weighting  of  evaluation  factors.  These  restrictions  against  numerical  proposal 
scoring  and  factor  weighting  are  the  greatest  contributors  to  cryptic  results  from  SSETs.  The 
recommendations  to  avoid  cryptic  SSET  results  and  distinctly  identify  the  best  value 
proposal  include  the  assignment  of  numerical  scores  for  subjectively  rated  factors  and  use 
of  proposed  numerical  values,  or  values  recalculated  for  reasonableness  when  appropriate, 
for  objectively  rated  factors.  The  numerical  weighting  of  evaluation  factors  is  also  included  in 
the  recommended  process  for  evaluating  proposals.  A  series  of  equations  was  developed  to 
convert  the  numerical  scores  to  a  common  basis  such  that  high  scores  are  favorable  for  all 
factors  and  to  weigh  the  scores  according  to  the  significance  of  the  factors  to  the 
government.  An  example  of  converting  numerical  scores  to  a  common  basis  wherein  low 
values  are  favorable  to  the  government  is  the  conversion  of  cost/price  to  render  low 
proposed  amounts  for  cost/price  to  high  scores.  Once  the  scores  are  converted  such  that 
they  can  be  compared  on  an  equivalent  basis  and  weighed  according  to  their  importance  to 
the  agency,  all  factor  scores  can  be  added  with  the  proposal  having  the  highest  combined 
score  being  the  proposal  offering  the  best  value  to  the  agency. 

The  values  recommended  here  for  subjectively  rated  factors  fall  in  a  range  of  70  to 
100  as  follows: 
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Score  range  Narrative  description  of  factor  rating 
90-100  Factor  meets  all  and  significantly  exceeds  numerous  expectations 

80-89  Factor  meets  all  and  exceeds  several  expectations 

70-79  Factor  meets  minimum  expectations 
One  advantage  of  using  a  range  of  scores  for  subjectively  rated  factors,  as  shown 
above,  rather  than  colors  or  adjectives,  is  that  proposal  evaluators  are  likely  to  discern  a 
hierarchy  of  ratings  for  individual  factors  within  the  adjectival  or  color  category  from  each 
contractor’s  proposals.  When  adjectival  or  color  ratings  are  mandated,  evaluators  must 
show  tied  scores,  such  as  outstanding  or  blue,  for  all  factors  that  “meet  all  and  significantly 
exceed  numerous  expectations.”  When  numerical  ratings  are  permitted,  discerned 
differences  can  be  portrayed  through  scores  ranging  from  90  to  100. 

In  addition  to  converting  cost/price  to  high  values,  certain  other  objective  criteria 
might  also  be  more  favorable  to  the  government  with  low  values.  For  example,  the  lowest 
temperature  in  which  a  system  can  operate  is  obviously  more  favorable  to  the  government 
when  lower  operating  temperatures  are  likely  to  be  experienced.  The  formula  developed  for 
converting  low  cost/price  amounts  to  high  values  and  weighing  the  values  to  reflect  the 
significance  of  such  factors  is  the  same  formula  for  converting  scores  for  factors  such  as  low 
operating  temperature.  That  formula  shall  be  referred  to  as  the  formula  for  “objective 
factor — low  numbers  are  favorable.”  A  complete  list  of  the  factor  types  for  which  equations 
have  been  developed  is  shown  in  Table  2,  Types  of  Factors  and  Formulae. 


Table  2.  Types  of  Factors  and  Formulae 


Type  of  Factors 

Formulae 

Subjective  factors  -  high  numbers  favorable 

S  =  W(R/HR) 

Objective  factors  -  high  numbers  favorable 

S  =  V/(HO/W) 

Objective  factors  -  low  numbers  favorable 

S  =  HO-(V-L)/(HO/W) 

Objective  factors  with  an  optimal  value 
(Ascending  section  of  line) 

(Descending  section  of  line) 

S  =  V/(OV/W) 

S  =  (OV  -  (V  -  OV))/(OV/W) 

Objective  factors  with  a  suboptimal  value 
(Descending  section  of  line) 

(Ascending  section  of  line) 

S  =  SV  -  (V  -  LVL)/(SV/W) 

S  =  SV  -  (HVL  -  V)/(SV/W) 

Where: 


HO 

= 

Highest  observed  value 

HR 

= 

Highest  possible  subjective  rating 

HVL 

= 

High  value  limit 

L 

= 

Lowest  observed  value 

LVL 

= 

Low  value  limit 

OV 

= 

Optimal  value 

R 

= 

Actual  subjective  rating 

S 

= 

Weighed  score 

SV 

= 

Suboptimal  value 

V 

= 

Observed  value 

W 

= 

Criterion  weight 

Illustrating  Identification  of  the  Best  Value  Proposal 

To  support  this  recommended  approach  in  the  face  of  potential  arguments  against 
numerical  factor  ratings  and  weighting,  a  similar  process  was  implemented  and  is  in  use  for 
evaluating  proposals  by  at  least  seven  states,  two  large  cities,  and  one  large  university. 


\\  '  t  )  <■;  ,S,|AM  fj 
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While  these  state  and  local  government  agencies  should  be  commended  for  using  an 
advanced  proposal  rating  schema,  their  process  contains  a  slight  flaw.  The  flaw  emanates 
from  an  anomalous  equation  used  for  converting  low  scores  to  high  values.  The  formula  for 
converting  objectively  rated  factors  where  low  values  are  favorable,  depicted  in  Table  2, 
does  not  contain  this  anomaly.  Although  the  anomaly  is  sufficiently  obscure  that  it  remained 
unnoticed  for  years,  it  could  result  in  selection  of  a  less  than  optimal  contractor  in  highly 
competitive  procurements.  Despite  the  flaw,  however,  their  process  is  considered  superior  to 
the  DoD’s  process  involving  scoring  matrices  that  display  a  confusing  patchwork  of 
adjectives,  colors,  and  numbers  that  oftentimes  fail  to  identify  the  proposal  offering  the  best 
value. 


Since  factor  weights  and  the  proposal  scoring  process  need  to  be  revealed  in  the 
solicitation,  it  is  not  possible  to  introduce  numerical  weighing  and  scoring  processes  after- 
the-fact  and  expect  to  identify  the  contractor  that  offered  the  superior  proposal.  The 
information  gleaned  from  the  GAO  report  with  respect  to  factor  weights  and  discerned 
differences  by  the  SSET  in  the  Boeing/Northrop  Grumman  case  can,  however,  be  converted 
to  numerical  values  to  demonstrate  how  the  recommended  methodology  for  weighing 
evaluation  factors  and  scoring  proposals  would  remove  ambiguity  from  the  SSET  results. 
Factor  weights,  in  this  example,  were  constructed  based  on  the  importance  of  the  factors  as 
described  in  the  solicitation.  The  adjectival  and  color  scores  were  converted  to  numerical 
scores  based  on  the  SSET’s  discernment  of  differences  in  the  proposal  scoring  as  depicted 
in  the  GAO  decision.  The  numerical  scores  were  then  weighed  according  to  their  importance 
and  low  numbers  were  converted,  when  appropriate,  to  high  values  so  that  all  factor  scores 
could  be  evaluated  on  an  equivalent  basis.  Table  1  was  then  reconfigured  to  reflect  the 
factor  weights  and  the  numerical  scoring  in  Table  3,  Revised  Proposal  Rating  with  Weighted 
Numerical  Ratings  for  Aerial  Refueling  Tanker. 
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Table  3.  Revised  Proposal  Rating  With  Weighted  Numerical  Ratings  for  Aerial 

Refueling  Tanker 


Boeing 

Northrop  Grumman 

Mission 

Capability/Proposal  Risk 

Factor 

Weight 

Numerical 

Rating 

Weighted 

Rating 

Numerical 

Rating 

Weighted 

Rating 

Key  System 
Requirements 

21 

88 

18.5 

98 

20.6 

System 

Integration/Software 

17 

80 

13.6 

80 

13.6 

Product  Support 

12 

92 

11.0 

92 

11.0 

Program  Management 

8 

88 

7.0 

98 

7.8 

Technology 

Maturity/Demonstration 

2 

80 

1.6 

80 

1.6 

Past  Performance 

30 

82 

24.6 

84 

25.2 

Cost/Price  (MPLCC) 

15 

SI  08.044 
Billion 

15.0 

$108,010 

Billion 

15.0 

Cost  Risk 

SDD  Phase/Production 
&  Deployment  Phase 

0 

Moderate/ 

Low 

N/A 

Low/Low 

N/A 

IFARA  Fleet  Effectiveness 
Value 

15 

1.79 

14.1 

1.9 

15.0 

Combined  Total  Score 

105.4 

109.8 

MPLCC  =  Most  Probable  Life  Cycle  Cost 

IFARA=  Integrated  Fleet  Aerial  Refueling  Assessment 

SDD  =  System  Development  and  Demonstration 

The  combined  total  score  of  109.8  for  Northrop  Grumman  in  Table  3  indicates  that 
this  contractor  best  meets  the  criteria  established  by  the  agency.  This  representation  of  the 
SSET’s  evaluation  of  proposals  offers  a  distinctly  clearer  identification  of  the  contractor 
offering  the  best  value  to  the  agency  than  the  Table  1  representation  that  reflects  virtually 
tied  scores,  leaving  the  SSA  with  flexibility  to  justify  award  to  either  contractor.  In  one  of 
those  rare  instances  where  the  SSA  is  subject  to  corruptibility,  the  present  method  for 
expressing  the  importance  of  evaluation  factors  in  relative  terms  and  scoring  factors  with 
adjectives  or  colors  facilitates  the  fraudulent  award  of  contracts. 

The  GAO  (2011a)  decision  regarding  the  sustaining  of  a  protest  filed  by  Solers, 
Incorporated  further  illustrates  problems  associated  with  identifying  the  contractor  offering 
the  best  value  to  the  government.  The  procuring  agency  in  the  Solers  case  was  the  DoD 
Defense  Information  Systems  Agency  (DISA).  Although,  as  with  the  Air  Force’s  aerial 
refueling  tanker  procurement,  the  GAO  sustained  the  protest  on  numerous  grounds  and  did 
not  specifically  identify  the  cryptic  SSET  results.  Indicators  contained  in  the  GAO  decision, 
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however,  further  support  a  process  to  more  definitively  determine  the  contractor  offering  the 
best  value  consistent  with  the  scoring  and  weighing  of  criteria  established  by  the 
government.  According  to  the  GAO  decision,  DISA  stated  the  importance  of  the  factors  in 
the  solicitation  as  follows: 

The  technical  factor  was  more  important  than  the  management  factor,  and  these  two 
factors  were,  combined,  more  important  than  the  two  past  performance  factors.  For 
purposes  of  award,  the  non-price  factors  were  “significantly  more  important”  than  price. 
(GAO,  2011a) 

The  matrix  reflecting  DISA’s  scoring  of  the  proposals  from  Solers  and  Booz  Allen 
Hamilton,  Inc.  (BAH)  is  provided  in  Table  4. 


Table  4.  Solers/BAH  Decision  Matrix 


SOLERS 

BAH 

Technical 

Blue/Low  Risk 

Green/Moderate  Risk 

Management 

Blue/Low  Risk 

Blue/Low  Risk 

Past  Performance  Confidence 

Satisfactory 

Substantial 

Past  Performance  Relevancy 

Relevant 

Very  Relevant 

Price/Cost 

$27,419,622.57 

$24,586,719.75 

The  GAO  (2011a)  decision  included  the  following  narrative  characterizing  the  CO’s 
decision: 


The  Contracting  Officer  (CO)  served  as  the  source  selection  authority  for  the 
procurement.  The  CO  found  that  “[although  the  Solers  proposal  was  rated 
technically  superior  to  BAH’s  proposal,  the  identified  strengths  do  not  warrant 
payment  of  an  approximate  1 2%  higher  price.”  . . .  The  CO  concluded  that 
BAH  should  be  selected  for  award  based  on  the  following  rationale:  “Based 
on  the  technical  sufficiency  of  the  [BAH]  proposal,  the  superior  past 
performance,  and  the  lower  cost,  the  Government  recommends  awarding  the 
contract  to  [BAH].” 

The  data  in  the  Table  4  decision  matrix  does  not,  however,  unequivocally  identify 
Solers  as  the  contractor  offering  the  best  value  proposal.  The  CO’s  conclusion  stated  as 
“the  identified  strengths  do  not  warrant  payment  of  an  approximate  12%  higher  price” 
[emphasis  added]  could  have,  based  on  the  decision  matrix  data,  just  as  well  have  favored 
award  to  BAH  by  stating  that  the  identified  strengths  do  warrant  payment  of  a  price  higher 
by  approximately  12%.  Recall  that  DISA’s  statement  regarding  the  importance  of  the  factors 
reads,  “For  purposes  of  award,  the  non-price  factors  were  “significantly  more  important”  than 
price.”  The  combination  of  adjectival  scores  and  relative  values  representing  the  importance 
of  the  evaluation  factors  results  in  a  decision  matrix  that  could  just  as  convincingly  support 
contract  award  to  either  BAH  or  Solers.  Had  DISA  used  numerical  scoring  and  weighting  of 
the  factors  in  this  case,  as  recommended  here,  the  decision  matrix  would  have  clearly 
identified  the  contractor  proposing  the  best  value  to  the  government. 

Additional  Recommendations 

Although  the  use  of  numerical  scoring  and  weighting  of  factors  is  recommended 
here,  it  is  not  recommended  that  the  government  rely  solely  on  numerical  scoring.  When 
evaluating  factors,  it  is  recommended  that  the  proposal  evaluation  process  be  documented 
contemporaneously  with  the  evaluation  of  proposals  through  use  of  a  form  similar  to  the  one 
illustrated  in  Figure  1 ,  Source  Selection  Evaluation  Team  Factor  or  Subfactor  Rating. 
Creating  narrative  records  contemporaneously  with  the  evaluation  of  each  factor  is  a 
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recommended  practice.  With  respect  to  documenting  the  proposal  evaluation  process,  GAO 
decisions  have  historically  included  statements  regarding  the  need  to  contemporaneously 
document  proposal  evaluations  similar  to  the  following: 

Next,  our  review  of  the  record  reveals  that  the  agency  failed  to  adequately 
document  its  resolution  of  weaknesses  and  risks  in  several  areas  of  the 
technical  evaluation.  Specifically,  under  the  platform  integration  sub-element 
and  size  and  weight  sub-element  of  the  design  approach  subfactor,  the 
evaluation  documents  contain  risks  and  weaknesses  that  were  attributed  to 
Raytheon’s  proposal,  were  not  resolved,  but  nonetheless  disappeared  from 
the  evaluation  record  without  contemporaneous  documentation.  Although  an 
agency  is  not  required  to  retain  every  document  generated  during  its 
evaluation  of  proposals,  the  agency’s  evaluation  must  be  sufficiently 
documented  to  allow  our  Office  to  review  the  merits  of  a  protest.  (GAO,  2013) 

Implementation  of  the  form  illustrated  in  Figure  1  will  also  address  other  weaknesses 
in  the  federal  government’s  acquisition  process.  The  weaknesses  were  discovered  during 
an  evaluation  of  protests  sustained  by  the  GAO  over  a  two-year  period  that  was  conducted 
in  preparation  for  writing  a  government  contracting  book  (Curry,  2014).  The  acquisition 
infractions  discovered  during  the  review  of  sustained  protests  addressed  through 
implementation  of  this  form  are  listed  following  Figure  1. 
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Source  Selection  Evaluation  Team  Factor  Rating 


Program  Number/Name 
RFP  Number/Name 


Contractor 

Factor  or  Subfactor  Number 


Factor  or  Subfactor  Title 


Description  of  Factor  or  Subfactor  from  the  RFP  (Attach  Sheets  as  Required) 


Acceptability  of  Proposal 
Was  the  Proposal  Submitted  on  Time? 


Yes 


No 


Does  the  Proposal  Meet  All  RFP  Requirements7 

if  not  is  the  Proposal  Unacceptable? 

were  All  contractors  Afforded  an  Opportunity  to  Address  Deficiencies? 


Were  All  Required  Negotiations/Discussions  Conducted9 

The  Proposal  is  Unacceptable  Because: 


Rating  Method 


Numerical 

Rating 


Adjectival  Rating  and  Definition 


Exceptional  Proposed  solution  meets  and  significantly  exceeds  all  agency  s  requirements 

Good  Proposed  solution  meets  all  and  exceeds  several  agency  requirements 


90-100 


Satisfactory 


Proposed  solution  meets  minimum  agency  requirements 


70-79 


Evaluation  Team  Member  s  Numencal  Rating  for  this  Factor  or  Subfactor 


Contractor's  Strengths  With  Respect  to  this  Factor  or  Subfactor 


Contractor's  Weaknesses  with  Respect  to  this  Factor  or  Subfactor 

Evaluation  Team  Members  Signature 

|  Date 

Evaluation  Team  Member  s  Pnnted  Name 

Figure  1.  Source  Selection  Evaluation  Team  Factor  or  Subfactor  Rating 

Proposal  Evaluation  Weaknesses  Addressed  Through  Figure  1,  Source 
Selection  Evaluation  Team  Factor  or  Subfactor  Rating: 

•  Failure  to  evaluate  proposals  according  to  the  terms  of  the  solicitation.  This 
weakness  is  addressed  by  repeating  the  description  of  the  factor  or  subfactor 
in  the  Description  of  Factor  or  Subfactor  From  the  RFP  section  exactly  as  it 
appears  in  the  solicitation. 


M*  OISTIAU 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-581  - 


•  Failure  to  document  evaluation  of  proposals  concurrently  with  the  proposal 
evaluation  efforts.  Completing  this  form  during  the  proposal  evaluation 
process  will  resolve  this  failure. 

•  Improper  rejection  of  proposals  that  were  submitted  on  time 

•  Failure  to  reject  proposals  not  submitted  on  time 

•  Failure  to  determine  whether  the  proposal  meets  all  solicitation  requirements 

•  Failure  to  document  determination  as  to  whether  the  proposal  was 
acceptable  or  unacceptable 

•  Failure  to  offer  all  contractors  the  opportunity  to  address  proposal 
deficiencies 

•  Failure  to  conduct  all  required  negotiations/discussions 

•  Failure  to  document  reason  for  determining  that  a  proposal  is  unacceptable 

•  Failure  to  evaluate  proposals  according  to  the  terms  of  the  solicitation  is  also 
addressed  in  the  Rating  Method  section  of  the  form  by  repeating  the  rating 
method  exactly  as  it  appears  in  the  solicitation. 

•  Failure  to  record  the  contractors’  strengths  with  respect  to  each  factor 

•  Failure  to  record  the  contractors’  weaknesses  with  respect  to  each  factor 

The  memorandum  from  the  Director,  Defense  Procurement  and  Acquisition  Policy 
mentioned  in  the  Introduction,  transitions  from  pre-solicitation  activities,  Chapter  2,  to 
evaluation  and  decision  process,  Chapter  3.  There  is,  however,  an  overlooked  critical  phase 
in  the  acquisition  cycle  between  pre-solicitation  activities  and  the  evaluation  of  proposals. 
This  activity  shall  be  referred  to  here  as  the  pre-proposal  phase.  While  the  contractors  are 
the  parties  primarily  engaged  in  activities  through  preparation  of  their  proposals,  DoD 
personnel  are  responsible  for  managing  pre-proposal  communications  to  ensure  that 
contractors  are  treated  equally  while  they  prepare  their  proposals.  Answering  questions 
directly  to  an  individual  contractor  in  response  to  a  seemingly  innocent  query,  could  give  the 
contractor  posing  the  question  a  competitive  advantage  over  other  contractors.  One 
example  of  such  a  competitive  advantage,  although  not  necessarily  innocent,  during  the  pre¬ 
proposal  stage  of  the  contracting  cycle  occurred  during  an  acquisition  by  the  Army  Corps  of 
Engineers  (GAO,  2011b).  In  this  instance,  an  Army  official  who  was  transitioning  to  the 
private  sector  was  offered  a  position  with  the  parent  company  of  a  contractor  that  was 
competing  for  a  contract  to  be  awarded  by  his  agency.  In  response  to  a  query  from  that 
subsidiary  company,  the  retiring  Army  official  advised  the  subsidiary  company 
representative  that  it  could  offer  a  price  lower  than  the  budgeted  price.  The  solicitation, 
however,  indicated  that  there  would  be  no  advantage  to  offering  a  price  lower  than  the 
budgeted  amount.  A  statement  in  the  solicitation  advised  prospective  contractors  that 
offering  a  lower  price  would  subject  their  company  to  a  penalty.  The  Source  Selection 
Authority  for  the  Army  Corps  of  Engineers  cited  the  lower  price  as  a  reason  for  selecting  the 
subsidiary  company  for  contract  award.  The  decision  was  protested  based  on  this 
communication  and  other  irregularities.  The  GAO  sustained  the  protest. 

The  recommended  method  for  managing  pre-proposal  communications  and  treating 
contractors  on  an  equal  basis  is  to  require  all  contractors  to  pose  questions  via  e-mail  to  one 
agency  official,  designated  in  the  solicitation,  by  a  specified  date  (e.g.,  two  weeks  prior  to 
the  date  that  proposals  are  due).  This  date  is  suggested  to  permit  the  agency  sufficient  time 
to  prepare  agency  responses  to  contractor  questions  by  a  certain  date  (e.g.,  approximately 
one  week  before  proposals  are  due).  It  is  further  recommended  that  the  agency  establish  a 
website  for  responses,  also  identified  in  the  solicitation,  where  all  contractor  questions  and 
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agency  responses  are  posted.  Implementation  of  these  recommendations  will  ensure  that  all 
contractor  questions  and  agency  responses  are  identical  and  available  to  all  prospective 
contractors  at  the  same  time. 

Conclusion 

The  DoD’s  source  selection  procedures  prohibit  the  use  of  numerical  values 
assigned  to  reflect  the  importance  of  the  proposal  evaluation  factors  and  prohibit  numerical 
rating  of  the  merits  of  proposals  submitted  by  prospective  contractors.  This  prohibition 
against  using  numerical  values  results  in  inconclusive  results  from  source  selection 
evaluation  teams  (SSETs)  and  a  failure  to  identify  the  contractor  offering  the  best  value,  and 
renders  the  DoD  source  selection  process  vulnerable  to  fraud.  The  failure  to  adequately 
manage  pre-proposal  communications  and  failure  to  adequately  document  the  SSET’s 
scoring  rationale  contemporaneously  with  proposal  evaluation  are  also  serious  weaknesses 
in  the  DoD’s  contractor  selection  process.  Implementation  of  the  recommendations  offered 
in  this  paper  to  address  all  the  noted  shortcomings  would  result  in  a  pragmatic  approach  to 
the  selection  of  government  contractors,  improve  the  probability  of  selecting  the  contractor 
offering  the  best  value  to  the  government,  and  lessen  the  present  vulnerability  to  contract 
fraud. 
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